1. Field of the Invention
This invention relates to transmission of digital information over a communications network. More particularly, this invention relates to routing in packet switching systems.
2. Description of the Related Art
The meanings of certain acronyms and abbreviations used herein are given in Table 1.
TABLE 1Acronyms and AbbreviationsCPUCentral Processing UnitFIFOFirst-in, First-outGUIDGlobally Unique IdentifierIPInternet ProtocolNICNetwork Interface CardNUMANon-Uniform Memory AccessOSOperating SystemPCI ExpressPeripheral Component Interconnect ExpressQoSQuality of ServiceQPQueue PairRSSReceive-Side ScalingTCPTransmission Control ProtocolUDPUser Datagram ProtocolWQEWork Queue Entry
In most network communication scenarios, the ports of a network interface card (NIC) pass data packets to and from the host to which the NIC is connected. In Internet Protocol (IP) packet forwarding, however, a network node with multiple ports, such as a node with a multi-port NIC or with multiple NICs, may forward at least some of the IP packets that it receives back to the network rather than delivering the traffic to the host. Such traffic is referred to as “IP forwarding traffic”. The host, which is typically a processor having multiple CPUs, e.g., a multi-core processor, is responsible, through the operating system and network driver, for managing packet flows, in particular the component of the flows that is represented by IP forwarding traffic.
Contemporary NICs support multiple receive and transmit descriptor queues (multi-queue). On reception, a NIC can send different packets to different queues to distribute processing among the cores. The NIC distributes packets by applying a filter to each packet that assigns it to one of a small number of logical flows. Packets for each flow are steered to a separate receive queue, which in turn can be processed by separate cores. This mechanism is generally known as “Receive-Side Scaling” (RSS). The goal of RSS and the other scaling techniques is to increase performance uniformly.
The filter used in RSS is typically a hash function over the network and/or transport layer headers—for example, a 4-tuple hash over IP addresses and TCP ports of a packet. One hardware implementation of RSS uses a 128-entry indirection table where each entry stores a queue number. The receive queue for a packet is determined by masking out the low order seven bits of the computed hash for the packet (usually a Toeplitz hash), taking this number as a key into the indirection table and reading the corresponding value.
IP forwarding traffic tends to be concentrated in one flow or a small number of flows in which the packets hash to the same CPU cores. Without some load balancing scheme, this tends to overload some of the CPU cores, while others are underutilized. Current methods for spreading the processing load among CPU cores rely mainly on manual configuration (which may be automated by performance-tuning scripts) for binding specific CPUs for certain tasks, e.g., one CPU for incoming traffic and another for outgoing traffic. Such manual ad-hoc solutions may cause loss of compliance with strict requirements to perform both outgoing and incoming traffic processing on the same CPU core or a range of CPU cores for streams that have the same RSS hash values.